Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach

摘要

Recent years have witnessed a resurgence of interest in video summarization.However, one of the main obstacles to the research on video summarization isthe user subjectivity - users have various preferences over the summaries. Thesubjectiveness causes at least two problems. First, no single video summarizerfits all users unless it interacts with and adapts to the individual users.Second, it is very challenging to evaluate the performance of a videosummarizer. To tackle the first problem, we explore the recently proposed query-focusedvideo summarization which introduces user preferences in the form of textqueries about the video into the summarization process. We propose a memorynetwork parameterized sequential determinantal point process in order to attendthe user query onto different video frames and shots. To address the secondchallenge, we contend that a good evaluation metric for video summarizationshould focus on the semantic information that humans can perceive rather thanthe visual features or temporal overlaps. To this end, we collect denseper-video-shot concept annotations, compile a new dataset, and suggest anefficient evaluation method defined upon the concept annotations. We conductextensive experiments contrasting our video summarizer to existing ones andpresent detailed analyses about the dataset and the new evaluation method.